Industrial manufacturing relies on automatic inspection of objects being manufactured. One form of automatic inspection that has been in common use for decades is based on optoelectronic technologies that use electromagnetic energy, usually infrared or visible light, photoelectric sensors (such as photodetectors), and some form of electronic decision making.
Machine vision systems avoid several disadvantages associated with conventional photodetectors. They can analyze patterns of brightness reflected from extended areas, easily handle many distinct features on the object, accommodate line changeovers through software systems and/or processes, and handle uncertain and variable object locations.
By way of example, FIG. 1 shows an illustrative embodiment of a vision detector 100 according to an illustrative embodiment for inspecting objects on a production line. A conveyor 102 transports objects to cause relative movement between the objects and the field of view (FOV) of vision detector 100. Objects 110, 112, 114, 116 and 118 are shown. In this example, the objects include exemplary features upon which location and inspection are based, including a label 120 and a hole 124. More particularly, the exemplary vision detector 100 detects the presence of an object by visual appearance and inspects it based on appropriate inspection criteria. If an object is defective (such as the label-less object 116), the vision detector 100 sends a signal via link 150 to a reject actuator 170 to remove the object (116) from the conveyor stream. An encoder 180 operatively related to the motion of the conveyor (or other relative motion) sends a signal 160 to the vision detector 100, which uses it to insure proper delay of signal 150 from the encoder count where the object crosses some fixed, imaginary reference point 190, called the mark point. If an encoder is not used, the delay can be based on time instead.
In an alternate example, the vision detector 100 sends signals to a PLC for various purposes, which may include controlling a reject actuator. In another exemplary implementation, suitable in extremely high-speed applications or where the vision detector cannot reliably detect the presence of an object, a photodetector is used to detect the presence of an object and sends a signal to the vision detector for that purpose. In yet another implementation, there are no discrete objects, but rather material flows past the vision detector continuously—for example a web. In this case the material is inspected continuously, and signals are sent by the vision detector to automation equipment, such as a PLC, as appropriate.
Basic to the function of the vision detector 100 is the ability to exploit the abilities of the imager's quick-frame-rate and low-resolution image capture to allow a large number of image frames of an object passing down the line to be captured and analyzed in real-time. Using these frames, the apparatus' on-board processor can decide when the object is present and use location information to analyze designated areas of interest on the object that must be present in a desired pattern for the object to “pass” inspection.
As the above-described systems become more advanced and available, users may be less familiar with all the settings and functions available to them.
Reference is made to FIG. 2 showing a block diagram of a vision detector 200 according to an illustrative embodiment. A digital signal processor (DSP) 201 runs software to control capture, analysis, reporting, HMI communications, and any other appropriate functions needed by the vision detector. The DSP 201 is interfaced to a memory 210, which includes high-speed random access memory for programs and data and non-volatile memory to hold programs and setup information when power is removed. The DSP 201 is also connected to an I/O module 220 that provides signals to automation equipment, an HMI interface 230, an illumination module 240, and an imager 220. A lens 250 focuses images onto the photosensitive elements of the imager 260.
The DSP 201 can be any device capable of digital computation, information storage, and interface to other digital elements, including but not limited to a general-purpose computer, a PLC, or a microprocessor. It is desirable that the DSP 201 be inexpensive but fast enough to handle a high frame rate. It is further desirable that it be capable of receiving and storing pixel data from the imager simultaneously with image analysis.
In the illustrative embodiment of FIG. 2, the DSP 201 is an ADSP-BF531 manufactured by Analog Devices of Norwood, Mass. An ADSP-BF53x would operate at a reduced level of performance compared to the ADSP-BF561 manufactured by Analog Devices of Norwood, Mass. In the illustrated arrangement, the Parallel Peripheral Interface (PPI) 270 of the ADSP-BF531 DSP 201 receives pixel data from the imager 260, and sends the data to memory controller 274 via Direct Memory Access (DMA) channel 272 for storage in memory 210. The use of the PPI 270 and DMA 972 allows, under appropriate software control, image capture to occur simultaneously with any other analysis performed by the DSP 201. Software instructions to control the PPI 270 and DMA 272 can be implemented by one of ordinary skill in the art following the programming instructions contained in the ADSP-BF533 Blackfin Processor Hardware Reference (part number 82-002005-01), and the Blackfin Processor Instruction Set Reference (part number 82-000410-14), both incorporated herein by reference. Note that the ADSP-BF531, and the compatible ADSP-BF532 and ADSP-BF533 devices, as well as the ADSP-BF561, have identical programming instructions and can be used interchangeably in this illustrative embodiment to obtain an appropriate price/performance tradeoff. The ADSP-BF532, ADSP-BF533 and ADSP-BF561 devices are substantially similar and have very similar peripheral support, for example, for the PPI and DMA.
The high frame rate desired by a vision detector suggests the use of an imager unlike those that have been used in prior art vision systems. It is desirable that the imager be unusually light-sensitive, so that it can operate with extremely short shutter times using inexpensive illumination. It is further desirable that it be able to digitize and transmit pixel data to the DSP far faster than prior art vision systems. It is moreover desirable that it be inexpensive and has a global shutter.
These objectives may be met by choosing an imager with much higher light sensitivity and lower resolution than those used by prior art vision systems. In the illustrative embodiment of FIG. 2, the imager 260 is an LM9630 manufactured by National Semiconductor of Santa Clara, Calif. The LM9630 has an array of 128.times.100 pixels, for a total of 12800 pixels, about 24 times fewer than typical prior art vision systems. The pixels are relatively large at approximately 20 microns square, providing high light sensitivity. The LM9630 can provide 500 frames per second when set for a 300-microsecond shutter time, and is sensitive enough (in most cases) to allow a 300-microsecond shutter using LED illumination. This resolution would be considered far too low for a vision system, but is quite sufficient for the feature detection tasks that are the objectives of the present invention. Electrical interface and software control of the LM9630 can be implemented by one of ordinary skill in the art following the instructions contained in the LM9630 Data Sheet, Rev 1.0, January 2004, which is incorporated herein by reference. In an illustrative embodiment, the imager 260 can also be an Aptina MTV024 imager which has an array of 752.times.480 pixels, for a total of 360960 pixels. This Aptina MTV024 imager has a reduced frame rate.
It is desirable that the illumination 240 be inexpensive and yet bright enough to allow short shutter times. In an illustrative embodiment, a bank of high-intensity red LEDs operating at 230 nanometers is used, for example the HLMP-ED25 manufactured by Agilent Technologies. In another embodiment, high-intensity white LEDs are used to implement desired illumination. In other embodiments, green and blue LEDs can be employed, as well as color filters that reject light wavelengths other than the wavelength(s) of interest.
In the illustrative embodiment of FIG. 2, the I/O module 220 provides output signals 222 and 224, and input signal 226. One such output signal can be used to provide a signal (150 in FIG. 1) for control of the reject actuator 170. Input signal 226 can be used to provide an external trigger.
As used herein an “image capture device” provides means to capture and store a digital image. In the illustrative embodiment of FIG. 2, the image capture device 280 collectively comprises a DSP 201, imager 260, memory 210, and associated electrical interfaces and software instructions. As used herein, an “analyzer” provides means for analysis of digital data, including but not limited to a digital image. In the illustrative embodiment, the analyzer 282 comprises a DSP 201, a memory 210, and associated electrical interfaces and software instructions. Also as used herein, an “output signaler” provides means to produce an output signal responsive to an analysis. In the illustrative embodiment, the output signaler 284 comprises an I/O module 220 and an output signal 222.
It will be understood by one of ordinary skill that there are many alternate arrangements, devices, and software instructions that could be used within the scope of the present invention to implement an image capture device 280, analyzer 282, and output signaler 284.
A variety of engineering tradeoffs can be made to provide efficient operation of an apparatus according to the present invention for a specific application. Consider the following definitions:
b fraction of the field of view (FOV) occupied by the portion of the object that contains the visible features to be inspected, determined by choosing the optical magnification of the lens 250 so as to achieve good use of the available resolution of imager 260;
e fraction of the FOV to be used as a margin of error;
n desired minimum number of frames in which each object will typically be seen;
s spacing between objects as a multiple of the FOV, generally determined by manufacturing conditions;
p object presentation rate, generally determined by manufacturing conditions;
m maximum fraction of the FOV that the object will move between successive frames, chosen based on above values; and
r minimum frame rate, chosen based on above values.
From these definitions it can be seen that
      m    ≤                  1        -        b        -        e            n        and      r    ≥          sp      m      
To achieve good use of the available resolution of the imager, it is desirable that b is at least 50%. For dynamic image analysis, n is desirably at least 2. Therefore, it is further desirable that the object moves no more than about one-quarter of the field of view between successive frames.
In an illustrative embodiment, reasonable values might be b=75%, e=5%, and n=4. This implies that m.ltoreq.5%, i.e. that one would choose a frame rate so that an object would move no more than about 5% of the FOV between frames. If manufacturing conditions were such that s=2, then the frame rate r would need to be at least approximately 40 times the object presentation rate p. To handle an object presentation rate of 5 Hz, which is fairly typical of industrial manufacturing, the desired frame rate would be at least around 200 Hz. This rate could be achieved using an LM9630 with at most a 3.3-millisecond shutter time, as long as the image analysis is arranged so as to fit within the 5-millisecond frame period. Using available technology, it would be feasible to achieve this rate using an imager containing up to about 40,000 pixels.
With the same illustrative embodiment and a higher object presentation rate of 12.5 Hz, the desired frame rate would be at least approximately 500 Hz. An LM9630 could handle this rate by using at most a 300-microsecond shutter. In another illustrative embodiment, one might choose b=75%, e=15%, and n=5, so that m.ltoreq.2%. With s=2 and p=5 Hz, the desired frame rate would again be at least approximately 500 Hz.
Having described the general architecture and operation of an exemplary vision system (vision Detector 200) that may support an HMI in accordance with an embodiment of this invention vision, reference is now made to FIG. 3, which shows a diagram of a Graphical User Interface (GUI) screen 300 for a Human-Machine Interface (HMI), interconnected with a vision detector (100) like that shown and described with reference to FIG. 1 above. The screen can reside on any acceptable HMI, including, but not limited to an Laptop Personal Computer (PC); Desktop PC, personal digital assistant or Notebook Computer (for example PC 194), cell phone, smart phone, or other appropriate device having an appropriate communication link (e.g. USB, wireless, network cable, etc.) with the vision detector (100). An appropriate HMI interface (described in connection with the above-incorporated-by-reference METHOD AND APPARATUS) interconnects with the exemplary vision detector's DSP to allow communication with the HMI. Note that the layout and menu contents of the illustrated screen 300 is exemplary, and a variety of layouts and menu items are contemplated in alternate embodiments. As described above, it is contemplated that the HMI is interconnected to the detector during setup and monitoring or testing. During normal runtime on a production line, the HMI may be disconnected and the detector freely operates various alarms, reject actuators (170) and other interconnected devices, while receiving optical inputs from illuminated objects and electronic inputs from line devices such as the encoder (180).
In this embodiment, the GUI 300 is provided as part of a programming application running on the HMI and receiving interface information from the vision detector. In the illustrative embodiment, a .NET framework, available From Microsoft Corporation of Redmond, Wash., is employed on the HMI to generate GUI screens. Appropriate formatted data is transferred over the link between the vision detector and HMI to create screen displays and populate screen data boxes, and transmit back selections made by the user on the GUI. Techniques for creating appropriate screens and transferring data between the HMI and vision detector's HMI interface should be clear to those of ordinary skill and are described in further detail below.
The screen 300 includes a status pane 302 in a column along the left side. This pane controls a current status box 304, the dialogs for controlling general setup 306, setup of object detection with Locators and Detectors 308, object inspection tool setup 310 and runtime/test controls 312. The screen 300 also includes a right-side column having a pane 320 with help buttons.
The lower center of the screen 300 contains a current selection control box 330. The title 332 of the box 330 relates to the selections in the status pane 302. In this example, the user has clicked select job 334 in the general setup box 306. Note, the general setup box also allows access to an item (336) for accessing a control box (not shown) that enables setup of the imager (also termed “camera”), which includes, entry of production line speed to determine shutter time and gain. In addition, the general setup box allows the user to set up a part trigger (item 338) via another control box (not shown). This may be an external trigger upon which the imager begins active capture and analysis of a moving object, or it may be an “internal” trigger in which the presence of a part is recognized due to analysis of a certain number of captured image frames (as a plurality of complete object image frames are captured within the imager's field of view).
The illustrated select job control box 330 allows the user to select from a menu 340 of job choices. In general, a job is either stored on an appropriate memory (PC or vision detector or is created as a new job. Once the user has selected either a stored job or a new job, the next button accesses a further screen with a Next button 342. These further control boxes can, by default, be the camera setup and trigger setup boxes described above.
Central to the screen 300 is the image view display 350, which is provided above the control box 330 and between the columns 302 and 320 (being similar to image view window 198 in FIG. 1). This display shows a current or stored image frame captured by the vision detector and, essentially, represents the vision detector's current field of view (FOV). In this example, an object 352 is approximately centered in the display. For the purposes of describing the illustrative embodiment, the exemplary object 352 is a bottle on a moving line having a main cylindrical body 354 having a narrowed upper cap section 356 with a series of graphics 358 thereon. Any acceptable object or pattern can be substituted herein and the relative motion between the object and the field of view can be generated by moving the objects, moving the vision detector (or moving its FOV) or moving both the objects and the vision detector. In this example, the object 352 is relative light in surface color/shade. While the background 360 is relatively dark (as depicted by dot shading), in general, there should exist sufficient contrast or shade differences between at least some portions of the object and the background to attain a basis for detecting and inspecting the object. However, it is contemplated that the object may be mostly dark and the background can be lighter in an alternate example.
As shown in FIG. 3, the object 352 is either a real-time image being returned from the vision detector under appropriate illumination or it is a stored image. In either case, the image in display 350 is the one upon which setup of the detector is performed. In this example, the object 352 is centered in the display 350 with background space on either side. In other examples, the object may be moved more closely to a side of the display, such as when detection and inspection are based upon internal features located at a distance from an edge.
Before describing further the procedure for manipulating and using the GUI and various non-numeric elements according to this invention, reference is made briefly to the bottommost window 370 which includes a line of miniaturized image frames that comprise a so-called “film strip” of the current grouping of stored, captured image frames 372. These frames 372 each vary slightly in bottle position with respect to the FOV, as a result of the relative motion. The film strip is controlled by a control box 374 at the bottom of the left column.
Reference is now made to FIG. 4 showing a partial view of the diagram of the GUI display of FIG. 3, detailing an image view and associated control box with a cursor having automatically placed an edge-detecting locator of predetermined size and angle on the image view, and an exemplary threshold bar and setting slider within the control box, according to the illustrative embodiment. After performing other general setup functions (see box 306 in FIG. 3), the user may set up the mechanism for detecting the object 352 using the vision detector that is used herein as an example of a “vision system.” The user clicks the setup detectors button 380 in FIG. 3 to access the control box 410. Within this box the user may decide which direction he or she wishes to have detection occur. The choices are machine or line-movement direction (typically horizontally or left-to right/right-to-left across the FOV) (button 450), cross direction (typically vertically or transverse to machine direction) (button 452) or angle direction (button 454). Once a direction is chosen for a main detector (note that additional directions may be chosen by accessing the control box 410 at a later time), the box 410 invites the user to click on a location in the object image, and that click generates a rectangular Locator ROI graphic 460 with an associated plunger 462 that fits to an adjacent edge of the object 352, as shown. A detailed description of an automated system and method for placing and sizing both Locators and Detectors is taught in commonly assigned U.S. Pat. No. 7,636,449, entitled SYSTEM AND METHOD FOR ASSIGNING ANALYSIS PARAMETERS TO A VISION DETECTOR USING A GRAPHICAL INTERFACE, the teachings of which are expressly incorporated herein by reference. The generalized threshold level is also set by the automated process based upon the overall difference between the light and dark pixels along the edge transition adjacent to the Locator 460. In brief summary, the threshold level determines how much transition along an edge or other feature is needed to turn the Locator “on.”
In this example, when the user “clicks” on the cursor placement, the screen presents the control box 410, which now displays an operating parameter box 412. This operating parameter box 412 displays a single non-numeric parameter bar element 414 that reports threshold for the given Locator.
Once an object has been located within a field of view using the detectors of FIG. 4, it is typically desirable to apply certain tools to the object to determine certain qualitative features (i.e. label misplaced, etc.) without having to set up too many individual parameters and tests. For example, the width of a particular object is desired to be set during the training phase to train the system for the appropriate object width during run-time analysis. A caliper tool can be implemented in accordance with a prior art system. However, the user is required to expend effort and judgment during training. Some users desire a more automatic-automated option. The virtual caliper, although extremely useful, requires extensive user input and manual adjustment of individual operating parameters during training of an image.